Learning Context-free Grammars: Capabilities and Limitations of a Recurrent Neural Network with an External Stack Memory

This work describes an approach for inferring De-terministic Context-free (DCF) Grammars in a Connectionist paradigm using a Recurrent Neu-ral Network Pushdown Automaton (NNPDA). The NNPDA consists of a recurrent neural network connected to an external stack memory through a common error function. We show that the NNPDA is able to learn the dynamics of an underlying push-down automaton from examples of grammatical and non-grammatical strings. Not only does the network learn the state transitions in the automaton , it also learns the actions required to control the stack. In order to use continuous optimization methods, we develop an analog stack which reverts to a discrete stack by quantization of all activations, after the network has learned the transition rules and stack actions. We further show an enhancement of the network's learning capabilities by providing hints. In addition, an initial comparative study of simulations with rst, second and third order recurrent networks has shown that the increased degree of freedom in a higher order networks improve generalization but not necessarily learning speed.